Reference: Langenau E, Pugliano G, Roberts W, Hostoffer R. Summary of ACOP (American 
College of Osteopathic Pediatricians) Program Directors’ Annual Reports for First-Year 
Residents and Relationships Between Resident Competency Performance Ratings and 
COMLEX-USA Test Scores. Electronic Journal of the American College of Osteopathic 
Physicians. 2010; 2(7). 



Summary of ACOP (American College of Osteopathic Pediatricians) Program Directors’ Annual 
Reports for First- Year Residents and Relationships between Resident Competency Performance 

Ratings and COMLEX-USA Test Scores. 



Erik E. Langenau, D.O.; Gina Pugliano, M.A.; William L. Roberts, Ed.D.; Robert Hostoffer, 

D.O. 



Abstract 



Context: The Accreditation Council for Graduate Medical Education (ACGME) and American 
Osteopathic Association (AOA) endorsed the use of competency-based assessment, with the 
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intention to improve health care administration [1,2]. High-stakes licensing exams, such as the 
Comprehensive Osteopathic Medical Licensing Examination - USA (COMLEX-USA), are used 
for such assessments. Residency programs, like those certified by the American College of 
Osteopathic Pediatricians (ACOP), have also developed competency-based assessment tools. 

The purpose of this study is to summarize ACOP Program Directors’ Annual Reports of First- 
Year Osteopathic Graduate Medical Education (OGME-l) pediatric residents from 2005 to 2009, 
and to investigate the relationship between the competency ratings of residents as reported on the 
ACOP Program Director’s Annual Report and COMLEX-USA scores. 

Methods: Data sets from the ACOP Program Directors’ Annual Reports and National Board of 
Osteopathic Medical Examiner’s (NBOME) COMLEX-USA series were merged and analyzed 
for relationships. COMLEX-USA Levels 1, 2-CE, 2-PE, and 3 scores for first-time test takers 
were used for analysis. 

Results: Program directors rated OGME-l pediatric residents’ performance on competency 
measures favorably. Competency ratings from the ACOP Program Directors’ Annual Reports 
were strongly correlated with each other (r = 0.86). Pediatric OGME-l residents performed 
better on COMLEX-USA Level 2-PE overall, and on the Humanistic Domain of COMLEX- 
USA Level 2-PE ( t = 4.17, p c.001), but not as well on COMLEX-USA Levels 1, 2-CE and 3 ( t 
= -5.89, -2.53 and -3.21) compared to all medical students who took the COMLEX-USA exam. 
ACOP’s Program Directors’ Annual Reports overall average competency ratings correlated with 
COMLEX-USA 1, 2-CE, 3 scores (r = 0.33) and Biomedical/Biomechanical Domain scores of 
COMLEX-USA Level 2-PE (r = 0.29). 
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Conclusions: For pediatric OGME-1 residents, competency ratings drawn from the ACOP’s 



Program Director’s Annual Report are statistically related to COMLEX-USA examination 
scores. 

Terms: COMLEX-USA, Competency Assessment, Pediatric Residents, ACOP 

Introduction 

As of January 2007, all American Osteopathic Association (AOA)-sponsored residency 
programs are required to assess each of the seven core osteopathic competencies for each 
resident[l]. These competencies include osteopathic philosophy and osteopathic manipulative 
medicine, medical knowledge, patient care, interpersonal and communication skills, 
professionalism, practice-based learning and improvement, and systems-based practice. 
Pediatric residency program directors from the Accreditation Council on Graduate Medical 
Education (ACGME) and AOA residency programs are therefore now required to administer 
competency-based assessments for each resident during training[2]. 



Using the seven competencies as a framework, various tools have been utilized for assessing 
resident performance: board certification examination performance[3], in-training examination 
performance[3-4], overall clinical performance [5], residency supervisor ratings[6], portfolio 
analysis[7]. Objective Structured Clinical Examination (OSCE)[8-ll], program director ratings 
[12-14], and program director annual evaluations [15- 16]. To monitor residents’ progress 
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throughout their graduate medical education, the American Osteopathic Association (AOA) 



requires that program directors complete an annual report for each osteopathic resident[16]. 
Starting in 2005, osteopathic pediatric residency program directors have been using the 
American College of Osteopathic Pediatricians (ACOP) Program Directors’ Annual Report. 

This competency-based assessment is used to evaluate resident progress throughout his or her 
training. 

In previous studies resident performance, as rated by program director or residency supervisors, 
was positively correlated with class rank[6], third year grade point average (GPA)[12], overall 
GPA[13], undergraduate OSCE performance [6], USMLE Step 1 scores[12], USMLE Step 2 
scores[13], and USMLE Step 2 CS Prototype scores[13]. In comparison, Balentine et al 
reported no precise correlations between performance in medical school and emergency 
medicine residency training [14]. 

Although a few studies have addressed relationships between medical school-based performance 
measures and the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) 
[17-20], we know of only two studies that have investigated the relationship between COMLEX- 
USA and residency performance. Cavalieri et al reported positive correlations between all 
COMLEX-USA cognitive examination scores (Levels 1, 2-CE and 3) and the American 
Osteopathic Board of Internal Medicine (AOBIM) examination scores (in-service examinations 
and board certification exam)[21]. Sevensma et al reported positive correlations between 
COMLEX-USA cognitive examination scores and in-service examination scores across five 
different residencies [22]. To our knowledge, no studies have investigated the relationship 
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between COMLEX-USA Level 2- Performance Evaluation (COMLEX-USA-USA Level 2-PE) 



and resident performance. Furthermore, no studies have investigated the relationship between 
COMLEX-USA examinations Level 1, 2, 3-CE and clinical performance measures of pediatric 
osteopathic residents. In our study, we will present a summary of the ACOP Program Directors’ 
Annual Reports from 2005 to 2009, and investigate correlations between program director 
ratings of residents for each of the seven competencies and COMLEX-USA examination scores 
(Levels 1, 2-CE, 2-PE, and 3). 



Methods 

Measures: 

COMLEX-USA Examinations: 

COMLEX-USA Level 1, 2-CE and 3 Computer-based Cognitive Examinations 

COMLEX-USA computer-based examinations are designed to assess the osteopathic 
medical knowledge of osteopathic physicians to practice medicine without 
supervision[23]. COMLEX-USA is constructed in the context of medical problem- 
solving which involves clinical presentations and physician tasks. COMLEX-USA 
cognitive exams are computer-based, administering items in machine format. Items are 
knowledge-based, written by content experts at the various levels of medical education. 
Raw scores are simply the number of items answered correctly which are then placed on 
a 3-digit or 2-digit standard score scale for reporting purposes. Scale cut-scores determine 
pass-fail decisions. A 3-digit standard score of 400 or a 2-digit standard score of 75 on 
COMLEX-USA Level 1 or COMLEX-USA Level 2-CE is required to pass the 
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examination. A 3-digit standard score of 350 or a 2-digit standard score of 75 on 
COMLEX-USA Level 3 is required to pass the examination. 

COMLEX-USA Level 2-PE Clinical Skills Examination 

COMLEX-USA Level 2-PE is a Standardized Patient (SP)-based clinical skills 
examination, where candidates pass through 12-stations, interviewing SPs who are 
trained to simulate different medical complaints [24-25]. Each station includes a 14 
minute doctor-patient encounter followed by 9 minutes to complete a written patient note 
(SOAP note). Based on blueprint content specifications, cases vary according to 
presenting symptoms (respiratory, cardiovascular, neuromusculoskeletal, gastrointestinal, 
and other symptoms), age, gender, and race/ethnicity. Cases also vary in clinical 
complaints that could be acute, chronic, or provide opportunities for health promotion or 
disease prevention. 

Candidates must pass two domains (Humanistic Domain and Biomedical/Biomechanical 
Domain) to pass the examination. Failure in either domain results in failure of the entire 
examination. Using criterion-referenced cut scores, a pass/fail decision is made for each 
domain. These pass/fail conclusions are based on standards defined by minimal 
competency for entry into graduate medical education. Several measures are used to 
score individual candidate performance in each of the domains (Table 1). 



Table 1. COMLEX-USA Level 2 PE 


Domains 


Domain 


Humanistic Domain 


Biomedical / Biomechanical Domain 
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Component 


Global Patient 


Data Gathering 


OMT 


Patient Note 


Scores 


Assessment Tool 


(History and Physical) 




(SOAP Note) 



The Humanistic Domain is comprised of the Global Patient Assessment Tool to assess 
the candidate’s listening skills, respectfulness, empathy, professionalism, ability to elicit 
information, and ability to provide information. 

The Biomedical/Biomechanical Domain is comprised of three weighted component 
scores: (1) data gathering, which reflects the examinee’s ability to obtain a medical 
history and perform a physical exam; (2) written patient note, which reflects the 
examinee’s written communication and ability to synthesize information, develop a 
differential diagnosis, and formulate a diagnostic and treatment plan; and (3) osteopathic 
manipulative treatment (OMT), which reflects the examinee’s ability to integrate 
osteopathic principles and utilize OMT. The OMT component is specifically scored in 
25% to 40% of the encounters. 

ACOP Program Director’s Annual Report 

For each resident, the ACOP Program Director’s Annual Report is used by the program 
director to answer questions related to resident contact information, scholarly and 
research activity, continuity panel size, overall progress, and resident competency ratings. 
The program director assigns a competency rating to each of his/her residents on each of 
the seven osteopathic medical competencies using a Likert type scale ranging from 1 
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(lowest) to 9 (highest). The specific questions for each of the seven osteopathic medical 



competencies are presented in Table 2. 



Table 2. Competency-based questions on the ACOP Program Director’s Annual Report 



OSTEOPATHIC PHILOSOPHY AND OMT 

1 . Demonstrates competency in the understanding and application of OMT where appropriate. 

2. Integrates osteopathic concepts into the medical care provided to patients as appropriate. 

MEDICAL KNOWLEDGE 

1. Demonstrates competency in the application of clinical medicine to patient care. 

2. Knows and applies the foundations of clinical and behavioral medicine. 

3. Demonstrates knowledge of accepted standards of care. 

4. Remains current with new developments in medicine. 

5. Participates in life-long learning activities. 

PATIENT CARE 

1. Gathers accurate, essential information from all sources. 

2. Demonstrates competency in the performance of diagnostic and treatment procedures. 

3. Provides health care services that includes preventative medicine and health promotion. 

INTERPERSONAL / COMMUNICATION SKILLS 

1 . Demonstrates effectiveness in developing appropriate doctor-patient relationships. 

2. Exhibits effective listening, written and oral communication skills. 

PROFESSIONALISM 

1. Demonstrates respect for patients / families and acts as their advocate. 

2. Adheres to ethical principles in the practice of medicine. 

3. Is sensitive to cultural diversity i.e. religion, age, gender, sexual orientation, and disabilities. 

4. Is cognizant of their own physical and mental health in order to effectively care for patients. 

PRACTICE-BASED LEARNING 

1 . Treats patients in a manner consistent with current and evidence based information. 

2. Performs self-evaluations of clinical practice. 

3. Understands research methods, medical informatics, and the application of technology. 
SYSTEMS-BASED PRACTICE 

1. Understands national and local health care delivery systems. 

2. Advocates for quality health care on behalf of patients. 

3. Practices cost effective medicine. 
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After being reviewed by the ACOP Graduate Medical Education Committee, the reports 
are forwarded to the AOA. Recognition for completing an osteopathic pediatric 
residency program requires an annual report from both the resident and program director 
for each training year. 



Design : 

The data were drawn from and combined from several sources. First, a list of all 
pediatric OGME-1 residents that were reviewed by the ACOP Graduate Medical 
Education Committee between 2005 and 2009 was compiled from the ACOP Program 
Directors’ Annual Reports. For our analysis purposes, the list was condensed to include 
training institution, medical school graduation year, and competency ratings assigned by 
the residency program directors. The competency ratings (1 for lowest; 9 for highest) 
are applied to several items within each of the following competency-based categories: 
Osteopathic Philosophy and OMT, Medical Knowledge, Patient Care, 
Interpersonal/Communication Skills, Professionalism, Practice-Based Learning, and 
Systems-Based Practice. 

Second, a linking variable was created and matched to the NBOME database to 
extract all COMLEX-USA scores, using SAS version 9.1. Reported information is 
provided only for aggregate groups, and confidentiality was assured by creating a linking 
variable with arbitrary values without personal identifiers. Institutional Review Board 
approval was received to analyze these data. 

The PROC FREQ and PROC MEANS commands in SAS were used to derive 
candidate scores (now OGME-1 Pediatric Residents) on COMLEX-USA Levels 1, 2-CE, 
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2-PE (including domain scores), and 3. For each cohort, mean scores were computed for 



statistical comparisons. 



Sample: 

The sample consists of 134 OGME-1 pediatric residents. This sample represents 
all OGME-1 pediatric residents with ACOP Program Directors’ Annual Reports that 
were submitted to the ACOP Graduate Medical Education Committee for review between 
2005 and 2009. The sample includes residents from 19 pediatric training institutions 
across the United States. Table 3 lists the number of residents per training program. 



Table 3. Number of ACOP Program Directors’ Annual Reports of Pediatric OGME-1 
Residents, by institution (N=134) 



Training Institution 


Residents 


% 


Maimonides Medical Center. NY 


24 


17.9 


Doctors Hospital, OH 


16 


11.9 


Oklahoma State Univ Med Center, OK 


16 


11.9 


Geisinger Health System, PA 


14 


10.5 


Good Samaritan Hospital. NY 


13 


10.5 


Henry Ford Hospital, MI 


13 


10.5 


Palms West Hospital, FL 


9 


6.7 


Richmond Medical Center, OH 


8 


6.0 


Newark Beth Israel, NJ 


5 


3.7 


Sparrow Hospital, MI 


4 


3.0 


Charleston Area Medical Center, WV 


2 


1.5 


St. John Health, MI 


2 


1.5 


Miami Children's Hospital, FL 


1 


<1 


Children’s Regional, NJ 


1 


<1 


Orlando Health, FL* 


2 


1.5 


Tampa General Hospital, FL* 


1 


<1 


Medical Center of Central Georgia, GA* 


1 


<1 


Walter Reed Medical Center, DC* 


1 


<1 


San Antonio Uniformed Services, TX* 


1 


<1 



*Not AOA-accredited 



Of the total OGME-1 residents, all but 2 students completed all three levels of the 
COMLEX-USA cognitive examination (Levels 1, 2, 3-CE), and 94 residents (69%) took 
the COMLEX-USA Level 2-PE examination. A minority of OGME-1 pediatric residents 
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entered a training program prior to the requirement to take COMLEX-USA Level 2-PE. 



Figure 1 is a description of the various sample sizes used in the analysis. 



Figure 



. OGME-1 Sample Si zes fro m the ACOP Program Directo rs’ An nual Reports 



OGME-1 

Residents 



took COMLEX 
Level 1 



took COMLEX 
Level 2-CE 



took COMLEX 
Level 2-PE* 



took COMLEX 
Level 3** 



*40 OGME-1 Pediatric Residents entered residency prior to the requirement to take COMLEX-USA Level 2-PE 

**2 OGME-1 Pediatric Residents had not taken COMLEX-USA Level 3 at the time of study analysis 



Results 

The average rating for each item is shown in Table 4. There is little variability in 
competency ratings, and average scores are typically 7.5. Score distributions are normally 
distributed for interpersonal skills, professionalism, practice-based learning, and systems-based 
practice. In comparison, competency scores for osteopathic philosophy and OMT, medical 
knowledge, and patient care tend to show more positive ratings. Adequate reliability is shown 
for the program directors’ competency ratings, consisting of 22 items (0.98). 
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Table 4. Descriptive statistics of competency rating scores for Pediatric OGME1 Residents 
(N=134) 



Component 


Question 


Mean 


SD 


Osteopathic Philosophy and OMT 


1 


7.44 


1.174 




2 


7.45 


1.161 


Medical Knowledge 


1 


7.16 


1.138 




2 


7.13 


1.093 




3 


7.23 


1.156 




4 


7.21 


1.164 




5 


7.34 


1.164 


Patient Care 


1 


7.42 


1.119 




2 


7.34 


1.070 




3 


7.49 


.948 


Interpersonal / Comm. Skills 


1 


7.75 


.961 




2 


7.72 


.923 


Professionalism 


1 


7.91 


.977 




2 


7.99 


.965 




3 


7.89 


.955 




4 


7.75 


.963 


Practice-Based Learning 


1 


7.37 


1.001 




2 


7.40 


.982 




3 


7.21 


.997 


Systems-Based Practice 


1 


7.10 


.920 




2 


7.50 


.987 




3 


7.10 


.900 



12 




The scatter plot in Figure 2 depicts the relationship between training institution and 
average competency ratings. Program directors tend to rate residents narrowly, in the direction 
toward the high end of the scale. 

Figure 2. Plot of final competency score by training institution 




Competency rating scores did not vary significantly based on academic year or whether 
or not the resident took the COMLEX-USA Level 2-PE. Although there are statistical 
differences (p<.001 for F-tests of ANOVA) in competency ratings as a function of training 
institution, the groups were not homogenous, so the robustness of the ANOVA is weakened and 
the analysis between training institution and competency score is limited. Correlational analysis 
indicates statistically significant relationships between competency scores and training program, 
but the correlations are weak and may lack practical significance. 
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Individual competency item ratings were computed as a single mean score for each 
competency area (medical knowledge, patient care, OMT, interpersonal skills and 
communication, professionalism, systems-based practice, and practice-based learning). In 
addition, an overall “average” score was computed, as the mean across all competency 
dimensions. The mean competency area scores are all significantly correlated within the ACOP 
Program Director’s Annual Report (see Table 5). In addition, individual competency scores 
within the categories also significantly correlate within the program directors’ ratings for all 
comparisons. For instance, data not shown indicates a strong correlation between 
Professionalism Question 1 and Interpersonal/ Communication Skills Question 1 (.86). The 
same item of Professionalism is also moderately correlated to Systems-based Practice Question 2 
(.76). Furthermore, the variable of training institution is correlated with the overall average of all 
competency areas within the reports, but it is not correlated with all individual components; 
although statistically significant, correlations are weak and may lack practical significance. 



Table 5. Correlation matrix for component scores within the ACOP Program Director’s Annual 
Report 





OMT 


MK 


PC 


IC 


P 


PBL 


SBP 


AVG 


TRAIN 


OMT 


1 . 00 ** 


















Medical Knowledge (MK) 


.792** 


1 . 00 ** 
















Patient Care (PC) 


.677** 


.829** 


1 . 00 ** 














Interpersonal/Communication 


.599** 


.681** 


.813** 


1 . 00 ** 












(IC) 

Professionalism (P) 


.637** 


.683** 


.766** 


.881** 


1 . 00 ** 










Practice-Based Learning (PBL) 


. 668 ** 


.847** 


.851** 


.737** 


.759** 


1 . 00 ** 








Systems-Based Practice (SBP) 


.656** 


.782** 


794 ** 


.693** 


.709** 


.883** 


1 . 00 ** 






Final average score (AVG) 


.823** 


9 14** 


.823** 


.865** 


.873** 


.923** 


. 886 ** 


1 . 00 ** 




Training Institution (TRAIN) 


.271 


.207* 


.138 


.169** 


.162 


.147 


.196* 


.180* 


1 . 00 ** 



**Correlations significant at the .01 level 
Correlations significant at the .05 level 
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For all comparisons, scores on the cognitive exam scores of the COMLEX-USA series 
(Levels 1, 2-CE, and 3) significantly correlate with resident competency ratings, except for the 
correlation between Professionalism and Level 3 exam scores (Table 6). The strongest correlations 
are between Level 2 scores and program director ratings of medical knowledge (.38) and practice- 
based learning (.38). 



Table 6. Correlations between COMLEX-USA cognitive exam performance and ACOP Program 
Director’s Annual Report average competency scores 





Level 1 


Level 2 


Level 3 


OMT 


.215* 


.266** 


.196* 


Medical Knowledge 


.273** 


.375** 


.310** 


Patient Care 


.233* 


.316** 


.292** 


Interpersonal/Communication 


.174* 


.268** 


.244** 


Professionalism 


.177* 


.235** 


.155 


Practice-Based Learning 


.271** 


.375** 


.342** 


Systems-Based Practice 


.261** 


.345** 


.289** 


Final Score 


.262** 


.351** 


.295** 



**Correlations significant at the .01 level 
♦Correlations significant at the .05 level 



For the COMLEX-USA Level 2-PE, pediatric OGME-1 residents’ scores show no 
difference in comparison to everyone who took the exam on measures within the 
Biomedical/Biomechanical Domain (t = 1.56). However, pediatric OGME-1 residents score 
significantly higher than the cohort on Humanistic Domain of COMLEX-USA Level 2-PE ( t = 
4.17). Examining average scores on Levels 1, 2-CE, and 3 of the cognitive exams, pediatric 
OGME-1 residents have slightly lower scores when compared to the cohort. Compared to the 
cohort, pediatric OGME-1 residents score lower on Level l(t= -5.89) with the greatest disparity 
and slightly lower on the Level 2-CE (t=-2.53) and Level 3 (t= -3.21). 
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As shown in Table 7, average competency ratings are significantly correlated with the 
Biomedical/Biomechanical Domain and average scores on Levels 1, 2-CE, and 3. The Level 2- 
PE Humanistic Domain score is not correlated with program director competency ratings. 



Table 7. Correlations between competency and COMLEX-USA Scores 





ACOP Program 
Director’s Annual 
Report Competency 
Scores 
(ACOP) 


COMLEX Average 
Cognitive Scores 
(CBT) 


COMLEX Level 2-PE 
Biomedical/Biomechanic 
al Domain Score 
(Level 2-PE BM/BM) 


COMLEX- Level 2-PE 
Humanistic Domain 
Scores 

(Level 2-PE Hum) 


ACOP Program Director’ s 
Annual Report 
Competency Scores 
(ACOP) 


1 








COMLEX- Average 
Cognitive Scores (CBT) 


.325** 


1 






COMLEX-Level 2-PE 
Biomedical/Biomechanical 
Domain Scores 
(Level 2-PE BM/BM) 


.291** 


.277** 


1 




COMLEX- Level 2-PE 
Humanistic Domain 
Scores (Level 2-PE Hum) 


.143 


.073 


.311** 


1 



**correlations significant at the .01 level (two-tailed) 

CBT: Computerized-based testing composite score (COMLEX-USA Levels 1, 2-CE, and 3) 



Examining group differences, the first-time pass rates of pediatric OGME-1 residents are 
higher than the cohort on the COMLEX-USA Level 2-PE overall, and for both the 
Biomedical/Biomechanical and Humanistic Domains. Pediatric OGME-1 pass rates are lower for 
Levels 1 and 3 of the cognitive exam series, and identical for the Level 2-CE. Comparisons of 
pass rates are presented in Table 8. 



Table 8. COMLEX-USA pass rates for pediatric OGME-1 residents and cohort 



Measure 


Pass rate 
Pediatrics 


Pass rate 
Cohort 


Cognitive Exam 

Level 1 


85.1 


91.1 


Level 2 


89.6 


89.6 


Level 3 


82.6 


91.0 


PE Exam 
Humanistic Domain 


98.9 


97.3 


Biomedical/Biomechanical Domain 


98.9 


96.3 


Overall PE 


98.9 


94.1 



Note: pass rates based on first-time test takers 
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Comments 



The ACOP Program Director’s Annual Report is used to evaluate competency-based resident 
performance, providing a framework for giving constructive summative feedback to residents 
and satisfying the AOA requirement for monitoring resident progress. As shown by positive 
residency competency ratings, low variability, and sufficient reliability, a large number of 
program directors judged the majority of pediatric OGME-1 residents as performing quite well. 
Although results from this study suggest that pediatric residents are indeed performing well, 
program directors may not fully understand how to use the tool properly. In addition, the rating 
scale may not adequately distinguish the full range of resident ability. It is also likely that 
residents beginning a residency program enter with a high level of ability. Perhaps the rating 
form could be enhanced with providing training materials for completing the forms, descriptors 
for each of the ratings, or anchors for each of the questions for increased distinction between 
residents on abilities at this level of graduate medical education. Assuring that the program 
directors understand the rating scale and are not acquiescent in their scoring is warranted. 
Regardless, the tool seems to assess a range of competencies for resident performance. 

Individual competency scores and aggregated mean competency scores for each category were 
significantly correlated with each of the seven competencies. For instance, a strong relationship 
was shown between Medical Knowledge 3 (Demonstrates knowledge of accepted standards of 
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care) and Patient Care 1 (Gathers accurate, essential information from all sources). This finding 
indicates a resident who gathers accurate and essential information from all sources demonstrates 
knowledge of accepted standards of care. Because there is moderate to large portions of shared 
variance between individual items, individual items may not be assessing unique skills, and there 
may be some redundancy between the individual questions. 

Pediatric OGME-1 residents scored significantly higher on the COMLEX-USA Level 2-PE 
Humanistic Domain, as compared to osteopathic cohorts. Conversely, OGME-1 residents scored 
slightly lower on the COMLEX-USA cognitive examinations (COMLEX-USA 1, 2-CE and 3) 
compared to osteopathic cohorts. It is possible physicians who exceed in interpersonal and 
communication skills pursue a career like pediatrics, where communication is valued as an 
essential component. More research is needed to investigate this possibility. 

COMLEX-USA 1, 2-CE and 3 scores were all significantly correlated with residency 
competency ratings. For all three examinations, the highest correlations were between Medical 
Knowledge, Practice-Based Learning and Improvement, and Systems-Based Practice. Multiple 
choice cognitive examinations like COMLEX-USA 1, 2-CE, and 3, have been shown to be 
useful in measuring medical knowledge and clinical reasoning but not necessarily for patient 
care or verbal communication[26]. Because these three competencies share a strong knowledge 
component, it would be reasonable to assume that each of these is positively correlated with 
multiple-choice cognitive examinations, as our data suggests. 



18 




While competency ratings were positively correlated with COMLEX-USA cognitive 
examinations, only the Biomedical/ Biomechanical Domain of the clinical skills examination 
was significantly correlated with the program directors’ competency ratings. Performance in the 
Humanistic Domain of the COMLEX-USA Level 2-PE did not correlate with program director’s 
ratings. It is possible that correlations could not be demonstrated because the skills measured by 
the Humanistic Domain score are too general. For instance, the Humanistic Domain has six 
subcomponents — one being “eliciting information” which may prove to correlate with 
Interpersonal/Communication competency ratings on the program directors’ report. Further 
analysis would be required to investigate the relationships between program director ratings and 
subcomponent scores of the COMLEX-USA Level 2-PE. Alternatively, perhaps the Humanistic 
Domain and competency scores did not correlate because each is measuring something 
inherently different. Several studies acknowledge the complexity of making correlations 
between medical student and resident performance given the heterogeneity of assessment tools 
being used[5-6], and there seems to be a great need for a more consistent approach to the use of 
assessment tools for assessing predictive validity. 



The competency of professionalism deserves special attention. Although significantly correlated 
with COMLEX-USA Level 1 and 2-CE scores, professionalism competency ratings showed only 
weak relationships with Level 2-PE scores. Multiple-choice cognitive examinations may not be 
well-suited or structured to assess professionalism, but why then was the clinical skills 
examination not strongly related to the program directors’ ratings of professionalism? Three 
explanations are plausible and deserve further attention. First, overall and domain scores are not 
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sensitive enough to detect relationships between exam scores and competency ratings, and 
further research could involve analysis of component and subcomponent COMLEX-USA Level 
2-PE scores. Second, perhaps all pediatric residents function at an extremely high level for 
professionalism, restricting the range of ratings and lowering correlations between scores. 

Third, perhaps an additional tool needs to be developed — a standardized tool that evaluates the 
clinical and professional competence of post graduate pediatric residents, not undergraduate 
medical students. For instance, professionalism may be better measured by ongoing assessment 
tools in clinical practice, such as portfolios. 

For this study, these one hundred and thirty four osteopathic pediatric interns represent all 
pediatric OGME-1 residents who have been reviewed by the ACOP Graduate Medical Education 
Committee between 2005 and 2009; therefore, results are limited to this population and should 
not be generalized to other residents or specialties. A limitation of this study is excluding many 
D.O. pediatricians who train in traditionally allopathic programs, where program directors for 
this group of residents did not complete the ACOP Program Director’s Annual Report. As the 
American Academy of Pediatrics (AAP) or American Board of Pediatrics (ABP) has not 
reported the number of osteopathic pediatricians in the workforce, three statistics have been 
reported: over 400 pediatricians are now members of the AAP’s Section on Osteopathic 
Pediatricians [27], 656 D.O.s were enrolled in ACGME-approved residency programs in 2008- 
2009 (representing 8.1% of all pediatric residents in allopathic training programs)[28], and 190 
allopathic pediatric residency spots were filled through the National Resident Matching Program 
(NRMP) match in 2009 [29]. This suggests that a large number of osteopathic pediatric interns, 
who are board certified by the American Board of Pediatrics (ABP) and not the American Board 
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of Osteopathic Pediatricians (ABOP), were excluded from this study. Another limitation of this 
study is that there are no inter-rater reliability measures between program directors. 

Conclusions 

Combining information provided by ACOP and NBOME allows for valuable, informative and 
insightful analysis — analysis of external measures to help investigate the relationship between 
scores of each organization’s tools for assessment. Data support four major study outcomes: 
first, program directors typically rate pediatric OGME-1 residents positively; second, 
independent competency ratings of the ACOP Annual Program Director’s Report are strongly 
related to each other; third, compared to cohorts, pediatric OGME-1 residents perform better on 
COMLEX-USA Level 2-PE and the Humanistic Domain of COMLEX-USA Level 2-PE and 
slightly worse on COMLEX-USA Levels 1, 2-CE and 3; fourth, program director report ratings 
correlate with COMLEX-USA 1, 2-CE, 3 and Biomedical/Biomechanical Domain scores of 
COMLEX-USA Level 2-PE. 
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